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ABSTRACT 

Principles for college outcomes assessment, as 
outlined in the literature, are reviewed* Drawing on the experience 
of the Virginia State Polytechnic Institute and State University 
(Virginia Tech) during a decade of state~mandated outcomes 
assessment, a number of salient issues are identified* Faculty and 
academic issues include these: faculty resistance to assessment and 
accountability; relationship of faculty development to assessment; 
maintaining continuity year“to“year ; and continuous assessment and 
resulting curricular adjustment in academic units* Administrative 
considerations concern: data dissemination; financial support for 
assessment activities; curriculum changes resulting from frequent 
assessment; student involvement in learning; placement testing; 
evaluating success in introductory courses; qualifications of 
assessment personnel; and cooperation between assessment, 
institutional research, and planning* One additional issue is that at 
the state and national levels there appears to be a trend toward 
standardization of testing and opinion surveys* The paper foresees 
such trends as: statewide student testing; increased use of numerical 
indicators; common satisfaction questions such as those in alumni 
surveys; use of peer studies; increased sophistication of 
disciplinary accreditation bodies; close scrutiny of innovations; and 
increased pressure for assessment from the business sector* (MSE) 
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Lessons Learned from a Decade of Assessment 



Abstract 

A summary of practical lessons learned from approximately ten years of 
doing assessment in an organized way is provided. The university 
involved was among the first of its size and complexity to be required to 
approach state mandated assessment along a model analogous to those 
used for institutional self studies, with heavy emphasis on the role of the 
department. The observations shared should be especially useful to 
individuals and/or institutions in the early stages of an outcomes 



assessment process. 
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Introduction and Perspectives 

In 198S the Virginia legislature required the state coordinating board to 
study how the assessment of student learning might be undertaken. An 
advisory committee was formed from representatives of many different 
colleges and universities, and one conclusion was that each institution 
would address assessment in its own fashion within certain guidelines. 
Virginia Tech, the largest and most diverse university in the state, chose 
an approach which was aimed at formative evaluation, with a primary 
focus on departmental assessment. 

Purpose 

In the intervening years a great deal has been learned about what to do 
and not to do, some of it coming from evaluations of the assessment 
program. The conclusions below tend to be practical and experientially 
based. The purpose of the paper is to share, in a summary form, some 
lessons learned from a decade of experience in the field which are not 
readily available from other sources. 

Literature Review 

There is a large and growing body of literature on assessment topics, but 
few resources of the kind of practical advice offered here. Two recent 
books offering useful examples are exceptions to this general rule: Making 
a Difference: Outcomes of a Decade of Assessment in Higher Education 
(Banta & Associates, 1993) and Assessment in Practice: Putting Principles 
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to Work on College Campuses (Banta & Associates, 1995). Both of these 
books include information gathered at Virginia Tech and other colleges and 
universities. The latter volume in particular is useful in addressing the 
issue of how to use assessment data for program improvement. 

A relatively recent set of outcomes assessment principles was developed 
hy a group affiliated with the American A^'Sociation for Higher Education 
(1992). Entitled “Principles of Good Practice for Assessing Student 
Learning,” it presents nine statements describing sound assessment 

practices: 

• The assessment of student learning begins with 
educational values. 

•. Assessment is most effective when it reflects an 
understanding of learning as multidimensional, 
integrated, and revealed in performance over 
time. 

• Assessment works best when the programs it 
seeks to improve have clear, explicitly stated 
purposes. 

• Assessment requires attention to outcomes and 
also and equally to the experiences that lead to 
those outcomes. 

• Assessment works best when it is ongoing, not 
episodic. 

• Assessment fosters wider improvement when 
representatives from across the educational 
community are involved. 

• Assessment makes a difference when it begins 
with issues of use and illuminates questions that 
people really care about. 
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• Assessment is most likely to lead to 
improvement when it is part of a larger set of 
conditions that promote change. 

• Through assessment, educators meet 
responsibilities to students and to the public. 



Banta et ai’s most recent book developed nine thematic chapters around 
each of the above principles and added one of its own: 

• Assessment is most effective when undertaken 
in an environment that is receptive, supportive, 
and enabling. 

An earlier but similar set of principles was developed by the National 
Association of State Universities and Land Grant Colleges (1988). This set 
has more of an institutional and statewide policy flavor to it, as one might 
expect from such an organization witnessing the national interest in 
assessment and accountability generally at the time it was produced. It’s 
Statement of Principles on Outcomes Assessment states the following: 

• Institutional, program, and student outcomes 
assessment should focus, primarily, on the 
effectiveness of academic programs and on the 
improvement of student learning and 
performance. 

• States and institutions should rely primarily on 
incentives rather than regulations or penalties to 
effect student outcomes assessment and foster 
improvement 

• Institutional programs for evaluation and 
assessment should be developed in collaboration 
with the faculty. 
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• Assessment requirements should permit colleges 
and universities to develop institutional 
programs and define indicators of quality 
appropriate to their missions and goals and 
consistent with state-wide objectives and 
standards. 

• Colleges and universities should be encouraged 
to use multiple methods of assessment for 
improving teaching and learning and 
demonstrating achievement. 

«' Requirements for assessment should be fiscally 
conservative and avoid imposing costly 
evaluation programs on institutions or state 
agencies. 

• Within an institution, assessment programs 
should be linked to strategic planning or 
program review, or to some comprehensive 
strategy intended to encourage change and 
improvement. 



A survey conducted during the spring of 1990 found colleges and 
universities purporting to be involved in student outcomes assessment but 
not being able to point to curricular or other changes that has occurred as a 
result (Muffo, 1992). Most found the “Statement of Principles” to be useful 
as guidelines, however. 

As of 1995, five years later, fully a fourth of those surveyed in another 
study (including 42 percent of two-year colleges) had not yet begun an 
assessment program and were not committed to doing so, while 43 percent 
were at the start-up stage (Steele, 1996). Approximately 30 percent were 
in the implementation or utilization/acceptance stages. None of the 144 
responding institutions described itself as being in the commitment stage. 
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defined as, “Reflecting integration of assessment into decision-making and 
change process, widespread faculty involvement. (13)” 



Recently Culver and Ridley (1995) developed their own set of principles of 
what not to do, requiring the reader to determine how best to go about 
assessment by reversing their advice. They report some very sound 
principles nonetheless: 

• Talk to the administration and make sure they 
understand that assessment is just a fad and that 
there is no reason to support it either with 
words or cash. 

• When sharing the results with others on campus, 
give no thought to the political fallout that may 
be caused by the information collected. 

• Point out as quickly and clearly as possible that 
the purpose of assessment is to root out those 
faculty who have gotten by on inferior teaching 
skills for the past several years. 

• Always assume that results deeply buried in the 
assessment report’s appendices will never be 
read or published in the newspaper because of 
the Freedom of Information Act. 

• To implement assessment policies and 
procedures on your campus, create a large 
committee of at least 40 individuals so that all 
possible insights can be gained before movement 
begins. 

• If no one on the faculty has asked to see the 
latest report on assessment, assume that they 
have no interest in assessment, wouldn’t 
understand it anyway, or are living the 
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unexamined academic life and are unworthy to 
receive your pearls. 

• Make sure that assessment information is 
interpreted apart from the context in which it 
was collected. This makes interpretation cleaner 
and less complex. 

• Never delegate to others the actual analysis and 
interpretation of data. 

• Rely exclusively on hard data and stay away 
from squishy, qualitative, touchy-feely data. 

• Always assume that lightening only strikes the 
computer next door and that your computer’s 
hard drive will never crash. 

• Rely heavily on standardized testing that has 
been developed outside the university and 
provides an objective view of student progress. 

• Always depend on slick PERT charts and 
checklists to manage your projects. 

• If you hear of something going on at another 
school, use it at your school without delay. 

• Depend on it that new software always works 
perfectly the first time. 

• So that longitudinal comparisons can be made, 
once measures have been put in place, they 
should remain forever. Change is a sign of 
weakness. 

• Never show up at faculty meetings lest faculty 
get the idea that assessment is connected with 
teaching and learning. 

• No attempt should be made to evaluate the 
assessment process on your campus. 
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While the development of “anti-principles” remains somewhat rare, the 
search for general principles seems to go on. As of February 20, 1996, for 
instance, a distinguished group of engineering educators were discussing a 
white paper entitled, “A Framework for the Assessment of Engineering 
Education.” At that point it was available on the American Society for 
Engineering Education Internet home page at 
http://www.asee.org/asee/announce/frameworkee/. 

Observations 

The most basic observation is that there are lessons that can be learned 
. and passed along; most of these apply in a variety of institutional settings. 

A brief summary of these follows: 

Faculty /Academic Issues 

• Faculty are going to resist assessment/accountability. It’s 
uncomfortable to be judged, especially on the achievement of others 
such as students. The process is time-consuming and can be frustrating 
at times; there are no easy answers. Faculty cannot completely control 
the process and results. Frequently in the end they are pleasantly 
surprised at what they discover. 

• Faculty (and others) will only accept the results of studies if they gather 
the data themselves, especially if the data are about their department 
or unit Consequently, faculty "ownership" of any data is one of the 
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most important factors. It's far better to have less-th an -perfect data 
gathering by departmental faculty than perfect data from somewhere 
else. Only when they trust the veracity of the data are they likely to act 
upon them. 

• Most assessment programs try to separate program assessment from 
that of individual faculty members, but often the discussion comes 
around to improvement of teaching by individuals. A major frustration 
soon arises in that there are few good ways in practice to evaluate 
teaching other than student evaluations which, while valid for a range 
of topics, are far from comprehensive but often are the only tool 
available. Faculty development and assessment cannot be artificially 
separated indefinitely. It is common to find overlapping interests 
where mature assessment programs exist. 

• Continuity is easily lost year-to-year even without substantial turnover 
of departmental leadership, including major committee leadership, 
which rarely remains the same beyond five years. It's therefore 
important to constantly emphasize continuity over time in planning and 
reporting. 

• "Assessment" and resulting curricular adjustments go on constantly in 
good academic units. Whenever possible, tie what is perceived to be an 
external assessment requirement to ongoing accreditation and/ or other 
internal efforts. They should be mutually supportive. 




10 






Administrative Issues 



• Data must be disseminated in an easy-to-understand manner in order 
to be effective. For example , a monthly or bi-monthly one-page 
newsletter with graphics and summary statements in bullet format can 
be quite effective in communicating with a large number of faculty and 
staff. More detailed back-up reports may be necessary for the small 
number of people desiring more information. 

• A little bit of financial support for assessment activities goes a long way 
towards changing attitudes. It’s less the amount of money than the 
principle that counts. When asking people to spend a lot of time on 
doing something that they often are not anxious to do, it helps a lot to 
be able to offer a few dollars to help them defray out-of-pocket costs of 
surveys or other activities. As one chair of a large department noted, 
one often gets asked to do things, but seldom does anybody offer to 
help out financially in even a small way. 

• Changes in the curriculum that are a result of assessment efforts often 
lead to improved student satisfaction, but not always. For instance, a 
highly regarded mathematics professor found that his students learned 
better than a comparative group when he employed interactive 
software in a calculus class. Despite their learning more, he received his 
lowest teaching ratings ever. It appears that the students preferred 
what they were used to doing: being passive recipients of lecture 
material rather than being active problems solvers as required by the 
software. 
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• There is strong evidence that student involvement in learning leads to 
improved learning. Technology is making it easier and cheaper to 
involve students in their own learning and to assess that learning. 

• When testing for placement in particular, home grown tests are always 
better than standardized ones, since those developing the tests are the 
same faculty as those conducting the classes. They know what is taught 
and what skills are needed to succeed in their curriculum. Similarly, in 
determining if a program has been successful in producing the kinds of 
graduates that it attempts to produce, a local examination is far better 
than a standardized one. The latter is simply a compromise among 
experts and seldom tests exactly what any individual program is trying 
to accomplish. In addition, standardized tests sometimes do not report 
sub-scores to allow feedback on areas of strengths and weaknesses 
and/or report scores to individual students only as opposed to the 
academic program attempting to measure student learning. 

• In beginning subjects such as mathematics, chemistry, biology, English, 
etc., the true mark of success is student performance in higher level 
courses in "hose or related subject areas. Tracking students over time, 
therefore, provides valid measures of success of earlier courses. This 
can be especially useful if the beginning courses are being taught in 
multiple sections using different modes of instruction. In the end, some 
would argue, the true measure of educational effectiveness is student 
performance in the career and in life in general. 
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• Assessment personnel often are from non-technical backgrounds and 
are selected because of perceived personal integrity and respect by 
other faculty. Such characteristics often become insufficient in more 
mature programs where technical skills are necessary in order for 
progress to be made. 

• 1 here is a need for strong cooperation between assessment, institutional 
research, and planning in order for all of the programs to be successful. 
In recent years this has led to combining the three areas under the 
same supervisor in order to improve the cooperative environment. (For 
a summan. of r^ated issues, see Mclaughlin, Muffo, and Calhoun, 

1995 .) ^ 



State/Regional/National Issues 

• There seems to be a drift at the state and national levels in the U.S. 
towards increased standardized testing and standardized questions on 
opinion surveys. An example of the latter are the required common 
questions on alumni surveys in Kentucky. This reflects another trend - 
what seems to be a constant and increasing desire on the part of policy 

makers to compare institutions and states to each other. 

I 

Future Trends 

What are the trends to watch out for in the next several years? Here are 
some predictions based on observations from a variety of sources. Assume 
that all will be used in institutional budgeting in some fashion or another. 

13 




Statewide student testing has been tried and discontinued in at 
least one state (New Jersey). Other states are piloting or planning to 
pilot statewide examinations of students; critical thinking skills seems to 
be the most likely area of interest in the beginning. 

Numerical indicators of various kinds have been developed in Texas, 
Florida, South Carolina, and other states; this trend is likely to 
accelerate. 



Common satisfaction questions, such as in the institutional alumni 
surveys done in Kentucky as mentioned above, are becoming more 
frequent. Colleges and universities will be held accountable for 
explaining why their former students, as an example, are less satisfied 
than those from their sister institutions. 

Peer studies will be used at the state level to compare colleges and 
universities across state lines; this will allow richer analyses based on 
institutional mission. 

Disciplinary accrediting bodies will continue to become more 
sophisticated in their assessment practices. Adoption of such measures 
should lead to more faculty and administrative interest in assessment. 

Innovations such as the use of technology in the classroom will be 
closely scrutinized. Cost-benefit and other such analyses will be 



common. 
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• The business sector more frequently will require evidence of sound 
assessment practices of all institutions, private as well as public, as a 
pre-condition for providing financial (gift and research) and other 
support such as student internship opportunities. 

Implications for Institutional Researchers 

Many institutional researchers are directly or indirectly, e.g., via 
supporting roles, involved in outcomes assessment, with governmental 
entities requiring more and more in recent years. This paper has shared 
some lessons learned at one large, diverse university about what does and 
does not work in implementing assessment. It also has presented some 
predictions as to future directions, almost all of which are driven from 
accountability concerns of groups outside of higher education. These 
observations, taken along with the principles available from other sources, 
provide a basis on which to build a sound and forward looking assessment 
program. 
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